This sets up CI to run on pull requests against this repo using all three configs that we distribute.

To make CI go (somewhat) fast and be economical, I am configuring it to use some different parameters compared to the base configs. See the new ci/prep_for_ci.py script but namely:

• Shorter sequence_len so that we can use A100-40GB instances
• Truncated training data set with only 1000 examples
• 2 epochs
• Hardcoded small val_set_size, eval_batch_size, micro_batch_size

Having struggled a bit lot to get the CI to run due to surprising interactions between these variables (especially the validation set sizes) I am somewhat reconsidering whether this is a good idea and if it would instead be preferable to eat slow/expensive CI for the sake of distributing configs that we "know" will work. I think my preferred order of operations is:

• Merge this with a baseline CI setup
• Update to the latest version of axolotl, transformers, etc.
• Revisit the question of exactly what we do in CI, as some of the issues that I have had may be fixed